Skip to content

feat(bigquery): allow WRITE_APPEND in createQueryArrowStream - #9648

Open
ohmayr wants to merge 3 commits into
feat/bigquery-arrow-max-resultsfrom
fix/bigquery-arrow-allow-write-append
Open

ohmayr wants to merge 3 commits into
feat/bigquery-arrow-max-resultsfrom
fix/bigquery-arrow-allow-write-append

Conversation

@ohmayr

@ohmayr ohmayr commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Context & Problem

createQueryArrowStream rejected writeDisposition: 'WRITE_APPEND'. Per the design doc discussion on write dispositions, Arrow should match query(), which reads back the whole destination table.

Summary of Changes

  • Remove the WRITE_APPEND guard. With an explicit destination, the stream reads the table through a read session, so it returns the rows already in the table plus the appended ones. The JSDoc says so.
  • Replace the rejection tests with a unit test (WRITE_APPEND reaches configuration.query on jobs.insert) and a system test (WRITE_APPEND returns the old and new rows; WRITE_EMPTY on a non-empty table fails and leaves it unchanged).

Verification

Unit tests pass (594), and the Arrow system tests pass against the real API (25 passing, 1 pending: the TIMESTAMP(12) test needs BIGQUERY_PICOSECOND_SUPPORT).

@ohmayr
ohmayr requested review from a team as code owners October 16, 2026 02:59
@product-auto-label product-auto-label Bot added the api: bigquery Issues related to the BigQuery API. label Oct 16, 2026
@ohmayr
ohmayr requested a review from shivanee-p October 16, 2026 03:00

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes the restriction and associated tests that rejected the WRITE_APPEND write disposition when creating an Arrow query stream. Specifically, the check for WRITE_APPEND in createQueryArrowStream_ has been removed, along with the corresponding unit and system tests. There are no review comments to address.

@ohmayr ohmayr changed the title fix(bigquery): allow WRITE_APPEND in createQueryArrowStream chore(bigquery): allow WRITE_APPEND in createQueryArrowStream Oct 16, 2026
@shivanee-p

shivanee-p commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Can you add a couple of tests confirming that WRITE_APPEND adds the expected rows and that the other writeDisposition values overwrites the table rows as expected since that's pointed out in the design doc as expected behavior?

@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-allow-write-append branch from 977c17a to 4d56cd1 Compare October 16, 2026 21:33
@ohmayr
ohmayr changed the base branch from main to feat/bigquery-arrow-destination-table October 16, 2026 21:33
@ohmayr ohmayr changed the title chore(bigquery): allow WRITE_APPEND in createQueryArrowStream feat(bigquery): allow WRITE_APPEND in createQueryArrowStream Oct 16, 2026
@ohmayr
ohmayr changed the base branch from feat/bigquery-arrow-destination-table to feat/bigquery-arrow-max-results October 16, 2026 21:52
@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-allow-write-append branch from 4d56cd1 to 8f3395e Compare October 16, 2026 21:52
@ohmayr
ohmayr added this pull request to stack #9597 October 16, 2026 21:53
Remove the guard that rejected WRITE_APPEND. With explicit destination
tables, the stream reads the destination through a read session, so a
WRITE_APPEND query returns the existing rows plus the appended ones.
This matches query(), which also reads back the whole destination
table.

Replace the rejection tests with a unit test that WRITE_APPEND reaches
the jobs.insert configuration, and a system test that WRITE_APPEND
returns existing and new rows while WRITE_EMPTY fails on a non-empty
table and leaves it unchanged.
@ohmayr

ohmayr commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@shivanee-p Added. The new system test checks that WRITE_APPEND returns the existing and the new rows, and that WRITE_EMPTY fails on a non-empty table without changing it. The destination-table test from #9601 covers WRITE_TRUNCATE (100 rows replaced by 10, reading back only the new ones) and the default disposition on a new table.

@ohmayr
ohmayr force-pushed the fix/bigquery-arrow-allow-write-append branch from 8f3395e to 1428bca Compare October 20, 2026 22:37
With WRITE_APPEND the destination table also holds rows from before
the query. Say so in the createQueryArrowStream JSDoc, as query()
behaves the same, and limit the read-session comment about row order
to a table the query just wrote.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigquery Issues related to the BigQuery API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants